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SYSTEM-STRATEGY ROI: AN INTRODUCTION 

Let’s say your school district is wrestling with a common problem: your elementary ELA passing rates 
are low and projected to get lower with the advent of Common Core or other more rigorous 
standards. At the same time, your district has faced budget cuts for several years, and next year’s 
budget will only hold steady. How can you increase student learning with the same dollars? 

During your strategic planning process, you decide to use a Return-on-investment (ROI) analysis to 
identify the most impactful and cost-effective programs related to elementary literacy. Your director of 
teaching and learning compares the ROI of three reputable professional development programs for 
teachers and recommends the program that (based on early national results) seems significantly more 
impactful than the other two despite having similar costs. You are poised to shift precious funds 
towards that approach. 

This seems like a rational and reasonable way to ensure you are getting the most learning from 
your scarce dollars. But of all the possible ways to improve elementary literacy, is this PD program 
really the best option? Might the district see exponentially more student achievement-for-the-buck 
from recruiting and retaining better literacy teachers, empowering stronger principals, changing 
compensation structures, or intensively remediating the least effective elementary literacy teachers? 
And at what point in the yearly churn of strategic planning and budgeting would these kinds of 
strategies be considered — and get input from the many departments involved? You pause before 
implementing the PD program to consider a wider range of possibilities. 

The Promise and Difficulty of ROI in Education Today 

Return-On-Investment analysis is a tool for improving resource efficiency — which is to say, 
improving the impact of your limited resources. Widely used in the business world, it compares the 
expected gains (returns) per unit of cost (investment) of a variety of potential actions. In recent years, 
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there has been growing interest in adapting this approach to education — sometimes called educational 
productivity, or academic-ROI. 1 ' 

Education leaders do not seek a monetary return on their investment, like business leaders do; they 
seek greater student learning, or other outcomes like student citizenship, higher graduation rates, or 
increased lifetime earnings and career options. They want to use their scarce dollars on what works 
best for students. 

Though districts have made great strides in this direction, most ROI analysis is still missing a big 
opportunity. Like in the elementary literacy example, districts often use ROI to evaluate programs 
and initiatives — comparing this PD program to that one, or debating whether to invest in iPads, 
after-school tutoring, or a new instructional management system. These appear as line items on a 
budget and are obvious targets in the yearly cycle of funding cuts and additions. 

But this process misses some of the biggest cost drivers in the district’s budget and never addresses 
some of the fundamental resource issues that may actually drive student success. For example, districts 
rarely analyze the ROI of different aspects of teacher compensation (like paying teachers for advanced 
degrees), workforce management strategies (like hiring and retention policies), or how schools group 
teachers with students and organize time in the school day. These strategies don’t appear as line items 
and often involve input from multiple departments. But changes to these fundamental uses of people, 
time, and money may affect far more students, more intensively — and deliver the highest ROI. 

Many districts’ strategic planning processes are not set up to consider these deeper issues. Planning 
happens in departmental siloes, often without an opportunity for the academic and finance departments 
to work together. Moreover, strategic planning often comes after key resource decisions have been made. 
It is not uncommon for schools to be required to submit their schedules and staffing plans in April while 
school improvement planning happens in May or August. Any ROI analysis that happens after schools 
have made their major resource decisions is unhelpful at best, and frustrating at worst. 

A Different Approach 

To maximize the power of ROI analysis, education leaders need a new approach — one we’re calling 
System-Strategy ROI. This approach starts with the fundamental student need to be addressed and asks 
not “Which program is better?” but “What resources will meet this need?” This means considering a 
wider range of options, including those that: 

• Span departmental boundaries (i.e., not just focusing on PD, but considering changes to 
hiring, staffing, or retention policies) 

• Include structural costs and strategies (i.e., compensation reform, student-teacher 
assignment policies, funding systems) 

• May not even exist in the district at present, but are considered promising practices elsewhere 


2 


(i.e., new school designs that extend the reach of excellent teachers through teacher teaming 
or use technology with flexible grouping) 

To make this shift, school systems must ensure that their strategic planning process can influence 
the way big resource decisions are actually made — holding budgeting and planning discussions 
before scheduling and staffing plans are submitted, for example. School system leaders can help their 
teams to take a System-Strategy ROI approach by structuring the planning conversation around five key 
steps: 

It is important to note that ROI — in any form — is not a magic formula. It cannot turn weighty and 


SYSTEM-STRATEGY ROI: FIVE KEY STEPS 


1. Identify the core need: What fundamental student performance need are we focusing on, 
and what’s our theory of change for addressing it? 

2. Consider a broad range of investment options: What are the investments we currently 
make to address this need, and what else could we do? 

3. Define ROI metrics and gather data: What are the relative returns (costs weighed against 
benefits) to the set of current/potential options? 

4. Weigh investment options: What other factors do we need to consider, in order to select 
from among the options? 

5. Make investment decisions: How can we free resources to do what we want to do? 


complex decisions into simple ones; any estimate of ROI relies in part on professional judgment and 
in part on our still imperfect estimates of cost and student achievement impact. However with the 
recent advent of teaching effectiveness data and improved data systems, our ability to estimate impact 
and cost objectively and accurately has grown tremendously. ROI is a powerful tool for adding 
structure, rigor, and data-backed evidence to the difficult decisions a school system must make on 
behalf of its students. Ultimately, the most important thing that distinguishes System-Strategy ROI 
from the typical ROI is that it seeks to answer this fundamental question: 

“How can we use all our limited resources strategically to improve 
student achievement and meet our goals?” 
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SYSTEM-STRATEGY ROI: THE PROCESS 


Laying the Foundations 

The first requirement for setting up a System-Strategy ROI process is to bring together the right players 
with the right mandate. The strategic planning process in most school districts is not currently set up 
to facilitate this. Often the superintendent, chief academic officer, and relevant academic teams work 
on a strategic plan and then hand it off to the chief financial officer and finance department to figure 
out how to align it with the budget. At this point, a CFO might use ROI analysis to choose among 
several investment options. 

But with the System-Strategy ROI approach, the entire district leadership must see strategic planning 
and budgeting as one continuous process. The CAO and the CFO should see themselves as partners, 
each bringing different expertise and tools to the conversation. The conversation must have a broad 
enough mandate to include possible re-evaluation of structural investments and a focus on the 
long-term time horizon — not confined to cost-cutting and adjusting the annual budget. And 
importantly, it must be timed to happen before de facto resource decisions have been made at the 
school level. 2 

Secondly, this interdisciplinary team needs a set of structured questions that will enable a 
system-wide, broad approach to ROI. The five key steps that we explore below serve as a framework for 
decision-making. At the same time, it’s important to establish a culture of tolerance for 
imperfection and ambiguity. At the end of the day, no district will have all the data it needs to make a 
perfect decision. Any ROI analysis will need to combine science, judgment, and professional expertise 
to make decisions that are based on the best available data. The goal is to empower a research and 
data-informed decision process that is deliberate, considers the right range of options, asks the right 
questions, and helps focus resources on key district priorities. 

FIVE KEY STEPS 

1. Identify the core need: What fundamental student performance need are we focused on, 
and what performance targets or outcome measures will we track? 

A good decision-making process always starts with clearly defining the decision at hand. In this case, 
districts need to identify their most crucial student performance need(s) — for example, elementary 
literacy, ELL student math performance, the achievement gap between various student groups, or 
improved graduation rates. 

After defining the performance need, districts must next define the performance target and the 
outcome measures that will define success in meeting that need. For example, if elementary literacy is 
a student need, our performance target might be 75% proficiency within three years as measured on 
the ELA portion of the state exam. By clearly identifying the problem and the goal, we can do a 
better job of identifying a more comprehensive set of the initiatives and policies to address the goal. 
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It is important to note that districts may have a variety of important goals related to student 
outcomes. These include improved attendance, higher graduation rates, college and career readiness, 
student citizenship, and lifelong productivity, among others. Districts may also invest in programs 
aimed at reducing teacher absences or fostering better communities. But while dropout prevention 
and improved student learning may both be important district goals, districts should focus on only 
one goal at a time when using an ROI approach. This is because an ROI approach cannot compare 
the relative return for initiatives that are aimed at different outcome measures. Any particular ROI 
analysis must focus only on those district goals that have comparable outcome metrics. For the 
purpose of this paper, we will focus on student-learning needs, as this is arguably the most important 
example of a return for which districts can create comparable impact metrics. 


QUESTIONS 


FUNDAMENTAL STUDENT NEED 


• What fundamental student need are we targeting? 


• What performance target do we want to reach, and what outcome measures will we use to 
gauge progress and success? 


2. Consider a broad range of investment options: What are all the investments we 
currently make to address this need, and what else could we do? 

Given the fundamental student need and performance targets — what potential investments could 
address them? This is where the district team has the opportunity to step back from the program- 
evaluation lens and consider a broader range of options that may have an even higher ROI. Leaders 
should create a theory of action that explains what’s driving the need and what would address those 
causes. For example, struggles with elementary literacy might be rooted in a gap in teacher skills, or 
perhaps ineffective teacher assignment or class sizes. To address these gaps, district teams using the 
System-Strategy approach should consider investments that: 

• Span departmental boundaries 

• Include structural costs and strategies 

• May not even exist in the district at present but are considered promising practices elsewhere 

If the district team thinks that teacher quality is an issue, it might consider professional development 
as a potential investment. But it should also consider hiring and retention policies that increase the 
number of excellent literacy teachers in the district; compensation structures that reward highly 
effective literacy teachers; or workforce management practices that extend the reach of excellent 
teachers within teacher teams or to more students. This means teams may consider deeply rooted 
structural costs that affect all teachers. For example, many districts spend millions of dollars on added 
pay to teachers who attain master’s degrees, despite evidence that such degrees are not correlated with 
increased student achievement. It may yield a higher ROI to reallocate that money to another 
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investment — and a good System-Strategy ROI analysis will highlight that fact. Similarly, districts 
should consider promising strategies that may not even exist in the district at the moment. 

In particular, there are many new ways to expand the reach of the most effective teachers or 
systematically improve the effectiveness of the overall teacher workforce over time. System-Strategy 
ROI requires looking past the line items on the budget and instead considering the fundamental uses 
of people, time, and money that contribute to student achievement. 


QUESTIONS 


CURRENT AND ALTERNATIVE OPTIONS 


• What is our theory of action for meeting this student need? What seems to cause the need? 

• What do we currently invest in this area or toward this end? 

• What people, time, and money do we currently invest in programs, instruction, and 
support targeted at related outcomes? 

• Do current investments align with our theory of action, and do we have evidence that they 
are working? 

• Are there things that are not working well? If we stop them, can we redirect those resources 
toward more productive options? 

• What broader alternatives can we imagine that would address the same set of student needs? 

• How does what we are currently doing support or interact with the proposed option? Are 
there dependencies we should be clear about? 

• What would have to be true of any new investment for it to be better than what we are 
currently doing? How likely is that scenario? 


3. Define ROI metrics and gather data: What are the relative returns (costs weighed against 
benefits) to the set of current/potential options? 

At this point, district teams are ready to determine the ROI of the various strategies identified in the 
previous step. At its core ROI is a simple concept, though estimating the components and factoring 
in data ambiguity can be quite complex. Consider the following approach to ROI described by Nate 
Levenson, which he calls, “a framework for making the thinking explicit and taking the emotional 
arguments out of it.” 3 - 

(Increase in Student Learning) x (# Students Helped) 

($ Spent) 

We will focus on each piece of the equation in turn, starting with student learning. 
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a. What is the likely return on student learning ? What amount of impact should 
we expect — and by when ? 

Student learning-impact estimates are at the heart of ROI analysis, but they are among the most 
difficult to define and interpret. Luckily, research and data advances in recent years are offering up 
more and more resources from which to pull information. 

Sources of Impact Estimates: 

Teaching effectiveness data: One of the most important new sources of impact estimates is teaching 
effectiveness data. Value-added or quantile growth performance data seek to describe how much a 
teacher contributed to students’ learning year by year, accounting for other factors. One of the 
benefits of TE data is that it is specific to each district, representing the real teachers and students at 
hand. With this data, districts can run simulations to estimate the effect on student learning of 
different human capital policies, such as: 

• Improving the professional growth or hiring practices of underperforming schools 

• Improving retention of top-performing teachers or expanding their roles 

• Remediating or managing out chronically underperforming teachers 

• Assigning teachers strategically to play to their strengths or to address student performance targets 

One of the drawbacks is that test-derived teaching effectiveness indicators are inherently imperfect 
estimates of a teacher’s value and can vary according to many factors; they should be used alongside 
other measures of teacher quality. 

Peer-reviewed research journals: Peer-reviewed journals can provide information about the 
predicted impact of a variety of potential policies. Peer-reviewed journals sometimes publish studies 
that include a control group and so can yield a more robust impact estimate. On the other hand, 
districts may not be able to implement the initiative with the fidelity of a controlled study and so 
must factor in the uncertainty of attaining the predicted impact. Additionally, there may be several 
studies which looked at the same strategy and found different effects on student learning. In 
choosing the right research to rely on, districts must use expert judgment and a conservative lens so 
as not to bias the result of the ROI analysis. 

Impact studies conducted within or without the school system: Districts will often supplement the 
national literature with their own impact studies of different initiatives. These can yield the most 
precise estimates of impact in a district context and are often used to expand a seemingly successful 
program or to cut a program that appears ineffective. While it is important for districts to measure 
the success of their own initiatives, it is often difficult or impossible to separate out the effect of the 
initiative being studied from other factors. At any given point, the visible ROI of an initiative may be 
quite different from its actual ROI or its potential ROI if implemented correctly. Districts must take 
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care in interpreting the results of impact studies, particularly where there is a small sample size or lack 
of a control group. 

Regardless of source, all impact must be used judiciously and placed in context. There are several 
factors that district teams should consider: 


• Magnitude and timing: How long is it estimated to take before there is impact on student 
learning, and how large is the impact? 

• Applicability: How similar is the model studied to your district context? 

• Implementation: How easy is it to replicate a particular strategy with fidelity to the way it 
was studied, or to scale it up? 

• Comparability of impact estimate: How easy is it to convert the outcome measure in the 
study (often standard deviation of test scores) to something useful for the district (i.e., likely 
impact on passing rates or year of student achievement)? 


In order to confidently analyze these many technical factors, it may be necessary for districts to bring 
in expert support in regressions, data analysis, and research study design. Such an outside eye can 
ensure the appropriate degree of conservatism in the estimates, urge caution around what the 
numbers can realistically show, and minimize bias. 


QUESTIONS 


MEASURING OR ESTIMATING IMPACT 


• What kind of data is available on the impact of the various policies, investments, or 
initiatives we identified? What’s the source? 


• How relevant is the data to our particular student performance need? Was the same 
question studied or an analogous one? How closely does it match our context, capabilities, 
and planned change? 

• How reliable were the observed outcomes? Was there wide variation in results across 
studies? Were they gold standard randomized controlled studies, small local pilots, or 
anything in between? 

• Did fidelity of implementation play a role in outcomes? 

• What are the mitigating or enabling factors? 

• Does this policy generate a one-time or a multiple-year impact on student learning? 

• How does combining various policies affect the expected impact of each on student 
performance? 

• How transferable are the observed outcomes to our context? What are the limitations? 

Or what could go better? 




b. How many students will this impact ? 

This is usually more straightforward than impact estimates, though not always. The first question to 
ask is what student sub-group(s) will benefit from the investment. District teams should deliberately 
consider the primary and secondary effects of any intervention. Are there students who are not 
targeted but will also be affected? For example, if you pay high-performing teachers to move to 
low-performing schools, many students will benefit, but some may also lose out. District teams 
should keep the overall impact to the system in mind when determining ROI. 


c. How much will it cost to fully fund? Over what period of time? 

The denominator of our ROI formula is cost. Though cost can also seem simple, district teams need 
to thoughtfully consider all the elements of cost over time. This includes: 

• Cost types: How much direct as well as indirect costs are required to achieve the desired 
outcome? How much will it vary across schools? 

• Magnitude and time frame of costs: How much one-time or start-up costs are required, and 
how much to sustain the strategy over time? Are there personnel or compensation costs that 
need to be considered? 

• Scalability: What are the fixed costs as well as variable costs, and how easy is it to scale costs? 

Scalability is an interesting factor because it varies so widely. Some policies or programs can be scaled 
at a flat cost per pupil with expectation of a similar performance impact. Other policies have a 
declining cost per pupil, while others may actually cost more per pupil, just to achieve the same 
improvement in outcomes. It is important to think through these dynamics when calculating costs. 

While it is rarely necessary or possible to be perfectly precise in cost estimates, it is important to 
clarify the degree of precision, the possible sources of error, and the extent to which different choices 
seem likely to affect quality of implementation and thus the expected performance impact. 


QUESTIONS 


COST FACTORS 


Start-up or transition cost: 

• What are the start-up/one-time costs required to initiate and transition to steady state? 

• How long will we need to invest above steady state and at what level to achieve the planned 
sustained student improvement? 


Ongoing or sustaining cost: 

• What is the annual recurring cost to sustain the initiative at steady state? (e.g., staff, 
materials, PD, regular updates) 

• Is it expected to rise or decline over time? 


CONTINUED 
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Indirect cost: 

• What else will need to be spent to achieve the desired result? 

• Is there infrastructure on which this initiative depends? Will that need to be upgraded? 

Note: It is usually most helpful to express cost on a per-impacted-pupil basis, using the number of 
students impacted from (b) above. 


4. Weigh investment options: What other factors do we need to consider, in order to select 
from among the options? 

Now the team needs to narrow in on a course of action: the most attractive combination of 
expected impact for the cost. Concurrently, you should step back out of the details to make sure 
you have considered the whole picture. This means putting your ROI analysis in the context of 
many other factors. 

a. Which option(s) have the highest likely return in our district context ? 

District teams should use the data, intuition, and judgment they have developed to compare their list 
of options using the ROI formula: return = the increase in student learning x the number of students 
impacted / by the total cost. When precise numbers are not available, it is useful to ballpark the ROI 
with estimates of low/medium/high, given what the teams do know about likely impact, number of 
students, and cost. It may be helpful to discount expected return in some way to account for the 
likelihood that a specific outcome will be achieved in your particular environment (either because the 
study data was slightly different or because implementation may be a challenge given local capacity). 

District teams should create a list of opportunities ranked in order of ROI, including a cost number 
associated with each option. But ROI is not a magic formula; it cannot choose the best option 
through math alone. System-Strategy ROI demands that district teams step back from the details and 
make sure the top options fit in the big picture context. That means considering these questions: 

b. How does this choice fit with our other district priorities, performance targets, 
and constraints ? 

Even if the district team has identified a high ROI option, it may in fact conflict with other 
established district priorities or potential new strategies — particularly if the investment spans multiple 
departments, touches on structural cost or uses of time, or is new to the district. It is important for 
teams to consider unintended consequences or costs, interactivity of this initiative with other 
actions in play, and the consistency of the initiative with the district’s overall theory of action. For 
example, in order to address the elementary literacy challenge, let’s say that your district decides to invest 
in expert coaches for teacher teams — a potentially high ROI strategy. At the same time, you must ensure 
that school schedules are structured to allow coaches to meet with teams on a consistent basis. By taking 
a System-Strategy approach — that is, iteratively considering many strategies as part of the same strategic 
planning process — the district team can ensure that all big investments fit together. 
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Additionally, the district teams may want to think about dosage across strategies. One strategy may 
yield a particularly high ROI when compared with other approaches to address that student need; but 
will that strategy alone reach the performance target? What combination of strategies must be 
considered to truly reach the goal? If a particular strategy has a high ROI, but also a high attendant 
cost, it may be a better use of resources to focus on a different student performance need. What other 
fundamental student performance need could the district address with those resources? 


QUESTIONS 


INTERACTIVITY AND CONSISTENCY 


• What could go wrong in our implementation that might affect outcomes? What might 
drive additional costs? 


• Have we considered the potential unintended consequences or “perverse incentives” of this 
particular course of action? 

• How might it impact other students or district priorities? (e.g. equity) 

• How does this fit with the district’s overall theory of action? 

• Is the cumulative impact of the chosen alternatives sufficient to meet our performance 
goals? Or do we need to consider additional or alternative investments? 


c. What is likely to happen if we don’t do this now ? 

It is important not to forget the last reasonable option: to do nothing or to choose a path of watchful 
waiting. District teams should consider the risk of doing nothing versus options to learn more and 
implement over time. 


QUESTIONS 


COST/ BENEFITS OF NO ACTION 


• What would happen if we took no action? Would we be worse off? 


• Could we make a more informed choice by waiting and gathering more data? What data 
would be most helpful? How would we get it? 

• Could we do something in the interim to gather more data or improve our chances of 
successful implementation? 


• Could we stage this investment in a way that would minimize our short-term outlay while 
providing additional data before we invest fully? 

• Could we get 80% of the impact in a more cost effective manner? Would that be a 
reasonable trade-off? 
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5. Make investment decisions: How can we free the resources to do what we want to do? 

Once the district team has decided on a desired course of action, they must plan how to reallocate 
resources to support the new initiative. How much will it cost, and how does that compare to the 
resources available to invest? How much additional would be needed to pursue the top option? 

Districts can and often do use ROI analysis to identify programs that can be cut to free resources for 
new, more promising initiatives. In fact, ROI analysis is designed to identify existing initiatives that 
have lower ROI than potential new options. But what if the new initiatives cost more than those the 
district plans to cut? 

Districts can often find additional resources to fund priority investments by finding ways to reverse 
misalignments — those accumulated spending patterns based on historical policies and practices that 
no longer achieve the necessary student impact results nor align with current strategic priorities. These 
resource misalignments typically fall into four categories as in the chart below. 


RESOURCE MISALIGNMENTS 


Piecemeal 

Missing a piece of a complex strategy 

For example: Creating collaborative 
planning time but not providing 
enough time, student data, or expert 
support for effective planning 


Under-investment 

Not investing enough in an effective 
strategy 

For example: Spending very little on 
recruiting and hiring excellent teachers, 
though that is a crucial way to increase 
effectiveness of teacher workforce 


Ineffective 

Tying up resources in ineffective strategies 

For example: Spending a high percent 
of teacher compensation dollars on 
things that are not tied to student 
improvement, rather than on factors 
like responsibility and results 


Over- investment 

Applying a promising practice to all, 
instead of a targeted few 

For example: Favoring across-the- 
board, small class size reductions 
rather than targeting small classes and 
extra time to students who are 
struggling in the fundamentals 
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After considering whether to cut the lower ROI initiatives from the analysis, districts can ask the 
following questions: Is the district pursuing high-cost strategies that are not shown in the literature 
to be effective (e.g., across the board class-size mandates that can’t get classes small enough to have an 
impact)? Or is the district investing a lot of money, per-pupil, in areas that may not be district goals? 
For instance, ERS analysis shows that districts often unintentionally create very small high school 
elective class sizes, even though basic proficiency in core classes may be a district priority. 

While it is beyond the scope of this article to provide instructions for how districts can conduct a full 
map of how their resource and spending patterns align with district priorities, several free resources to 
help districts identify strategic resource misalignments are available on the ERS website: 

School Budget Hold’em : A tool to help teams identify trade-offs they can make and the kind 
of savings that could be redirected to new investments for different types of decisions. 

Resource Check A tool designed to help district leaders and others analyze a district’s current 
resource use and highlight where it aligns (or does not) with best practices. 

Practical Tools for District Transformation : Detailed resource guides that provide 
suggestions for analysis and new strategies in a number of key areas: School Funding Systems, 
Turnaround Schools, School Design, The Teaching Job . 4 

Conclusion 

Making strategic investment decisions in school districts can feel daunting. The questions are big, 
the players are many, and resource allocation and planning processes can be messy and uncertain. 

But despite all of that uncertainty, applying rigorous logic to the resource allocation process can 
help leaders make more effective resource decisions. And when districts take the time to consider a 
System-Strategy perspective, ROI can become an effective part of a comprehensive decision process 
that helps leaders make the best use of their limited resources. 
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Getting there 

f While there is no single path to educational transformation, 

ERS' School System 20/20 provides direction, together with 
tools and publications, to help align school system resources to 
strategic priorities, so that every school succeeds for every student. 
Return on Investment in Education, A "System-Strategy" Approach is 
one of many publications designed to guide district leaders toward that goal. Based 
on experience with school systems across the country, School System 20/20 identifies 
seven key areas for transformation and documents the specific policy and resource 
use changes critical to each. The framework includes assessments to help district 
leaders evaluate performance and track progress toward success for all. 

For more information go to www.system2020.org. 
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